Skip to content

Wallet schema v2 #2146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
May 18, 2025
Merged

Wallet schema v2 #2146

merged 15 commits into from
May 18, 2025

Conversation

ekzyis
Copy link
Member

@ekzyis ekzyis commented May 2, 2025

Description

based on #2092 because it's the target branch

This PR does the following:

  • add static table WalletTemplate which stores which send and receive protocols a wallet in UserWallet supports
    => allows multiple send and receive protocols per wallet
  • split wallets by send and receive for easier validation of configuration instead of having both send and receive optional and rely on application logic to make sure the configuration is not invalid
  • add wallet protocol tables (WalletSend..., WalletRecv...) which store configuration per user wallet
  • use triggers to make sure that a protocol of a user wallet is actually supported by the wallet

How to add a new wallet:

  • add a new row to the WalletV2 table and declare which send and receive protocols it supports
  • if it supports a new protocol, update enums and add a new wallet protocol table to store configuration

My beautiful TODO list

  • use unique index for walletId in wallet protocol tables so a user wallet can't have the same protocol twice for the same wallet ✅
  • check if HTTP protocols to send and recv (LNbits, Blink, Phoenixd) can be abstracted away via one WalletSendHTTP table without resorting to JSON columns ❌ won't do in this PR, maybe in the future
  • check if trigger to check wallet support can be unified ✅
  • wallet migration ✅
    • wallet seed ✅
    • eliminate duplicate wallets before migration (Delete duplicate wallets #2163) ✅
    • migrate wallets ✅
    • wallet seed with multiple wallets (LNbits, NWC, ...) per user ✅
    • put send/receive protocols into the same user wallet if exists ✅
    • verify wallet migration with SQL assertions after migration ❌ decided not to, too much work
    • add trigger to update JSON in ProtocolWallet row ✅
      • split JSON column into two columns for send and receive? ❌ no longer needed since we switched to ProtocolWallet
      • resolve vault entries in JSON trigger ✅
    • drop old tables, triggers etc. ✅
    • keep ids stable during migration to make new foreign keys work ✅
  • figure out why Withdrawal foreign key fails ✅ fixed it, but didn't really figure out what happened
  • Q: Are there more edge cases with our old wallets that might break migration? ✅ No, I don't think so.
  • Q: add priority to ProtocolWallet, too? ✅ No, I don't think that's important and we can change it later.
  • verify wallet support 🚧
  • update code to save/load wallets 🚧
  • rename walletId foreign keys that point to UserWallet to userWalletId to avoid confusion? 🤔
  • add User(id, vaultKeyHash) foreign key to Vault table with new userId, keyHash columns? ❌ out of scope of this PR

Additional Context

This is based on the latest discussion with @huumn how to plan further ahead with the wallet schema than to just migrate the vault in #2092.

Checklist

Are your changes backwards compatible? Please answer below:

tbd

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:

tbd

For frontend changes: Tested on mobile, light and dark mode? Please answer below:

n/a

Did you introduce any new environment variables? If so, call them out explicitly here:

no

@ekzyis ekzyis added the wallets label May 2, 2025
@ekzyis ekzyis marked this pull request as draft May 2, 2025 06:11
@huumn huumn mentioned this pull request May 5, 2025
13 tasks
@ekzyis ekzyis force-pushed the wallet-refactor-migrate-vault branch from 7efb350 to a10af10 Compare May 5, 2025 23:05
@ekzyis ekzyis force-pushed the wallet-schema-v2 branch 3 times, most recently from eccc4d0 to 840bb20 Compare May 7, 2025 05:35
Copy link

gitguardian bot commented May 7, 2025

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
16991000 Triggered Generic High Entropy Secret c52b973 docker/db/wallet-seed.sql View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@ekzyis ekzyis force-pushed the wallet-schema-v2 branch 5 times, most recently from 36e027f to 84a7dcd Compare May 11, 2025 04:21
@ekzyis ekzyis force-pushed the wallet-refactor-migrate-vault branch from a10af10 to 020e297 Compare May 13, 2025 17:55
@ekzyis ekzyis force-pushed the wallet-schema-v2 branch from d58cca2 to af9311a Compare May 13, 2025 17:55
@ekzyis ekzyis force-pushed the wallet-schema-v2 branch 2 times, most recently from 9a6ce22 to c2a22f6 Compare May 15, 2025 04:57
@ekzyis ekzyis force-pushed the wallet-refactor-migrate-vault branch from 020e297 to e604617 Compare May 15, 2025 22:58
@ekzyis ekzyis force-pushed the wallet-schema-v2 branch 7 times, most recently from 6c07696 to e60046f Compare May 16, 2025 02:38
@ekzyis ekzyis force-pushed the wallet-refactor-migrate-vault branch 2 times, most recently from 310f17c to e604617 Compare May 16, 2025 22:12
@ekzyis ekzyis changed the base branch from wallet-refactor-migrate-vault to wallet-v2 May 18, 2025 02:46
@ekzyis ekzyis marked this pull request as ready for review May 18, 2025 03:59
@ekzyis
Copy link
Member Author

ekzyis commented May 18, 2025

See #2092 (comment)

@ekzyis ekzyis force-pushed the wallet-schema-v2 branch from 0e387a4 to 1637f1d Compare May 18, 2025 04:00
@ekzyis ekzyis merged commit 3a5324c into wallet-v2 May 18, 2025
6 checks passed
@ekzyis ekzyis deleted the wallet-schema-v2 branch May 18, 2025 05:00
@ekzyis ekzyis mentioned this pull request May 18, 2025
ekzyis added a commit that referenced this pull request May 20, 2025
ekzyis added a commit that referenced this pull request May 21, 2025
ekzyis added a commit that referenced this pull request May 23, 2025
ekzyis added a commit that referenced this pull request May 25, 2025
ekzyis added a commit that referenced this pull request May 31, 2025
ekzyis added a commit that referenced this pull request Jun 3, 2025
ekzyis added a commit that referenced this pull request Jun 4, 2025
ekzyis added a commit that referenced this pull request Jun 6, 2025
ekzyis added a commit that referenced this pull request Jun 6, 2025
ekzyis added a commit that referenced this pull request Jun 11, 2025
ekzyis added a commit that referenced this pull request Jun 13, 2025
ekzyis added a commit that referenced this pull request Jun 15, 2025
ekzyis added a commit that referenced this pull request Jun 15, 2025
ekzyis added a commit that referenced this pull request Jun 18, 2025
ekzyis added a commit that referenced this pull request Jun 18, 2025
ekzyis added a commit that referenced this pull request Jun 19, 2025
ekzyis added a commit that referenced this pull request Jun 20, 2025
ekzyis added a commit that referenced this pull request Jun 20, 2025
ekzyis added a commit that referenced this pull request Jun 26, 2025
ekzyis added a commit that referenced this pull request Jun 27, 2025
ekzyis added a commit that referenced this pull request Jul 6, 2025
ekzyis added a commit that referenced this pull request Jul 8, 2025
huumn added a commit that referenced this pull request Jul 15, 2025
* Migrate vault entries to new schema (#2092)

* Migrate existing vault entries to new schema

* Read+write new vault schema

* Drop VaultEntry table

* Refactor vaultPrismaFragments

* Remove wrong comment

* Remove TODO

* Fix possible race condition on update of vault key

* Remove lib/object.js

* Wallet schema v2 (#2146)

* Add wallet-v2 TODOs

* Update checkWallet

* Wallet list

* Delete almost all wallet v1 code

and add some code for wallet protocol forms

* Define protocol display name in JSON

* Show form per protocol

* Increase max-height of image in form

* Add JSdoc for protocols, form validation

* Use wallet cards again

My wallet list was quite ugly and I couldn't look at it anymore.

* Refactor hooks in wallet provider

* Fix PasswordInput not used

* Read encrypted wallets

* Decrypt wallets

* useWalletQuery now returns decrypted wallets
* Refactor useIndexedDB because its only purpose will be to store the key, so no need for pagination code etc.
* There is still a bug: if the wallet is not decrypted on first render, the form will not see the decrypted value. See TODO.

* Rename protocolJson to protocol

it no longer uses a JSON file

* Fix form not updated with decrypted API key

* Fix wallet template forms

* Fix optional shown as hint

* Rename to mapUserWalletResolveTypes

* Save LNbits send and recv

TODO:

* implement resolvers for other protocols
* fix double update required for trigger?
* add missing validation on server
* add missing network tests
* don't import from wallets/client on server

* Move definitions to lib/wallets.json and lib/protocols

* Fix ProtocolWallet.updated_at not updated by trigger

* Move wallet fragments into wallets/client/fragments/

* move invoice fragments to fragments/invoice.js
* remove some unused fragments that I don't think I also will not use
* move fragments that will be generated in own file

* Move wallet resolvers into wallets/server/resolvers

* Fix missing authorization check on wallet update

* Run all shared code in generic wallet update function

* Fix 'encrypt' flag not set for blink send currency

* Add mutations for all protocols

* Fix macaroon validation

* Fix CLN socket value not set

* Add server-side schema validation

* Fix JSDoc typedef for protocols

* Don't put JSDoc into separate file

* Create test invoices on save

* Also move type resolvers into wallets/server/resolvers

* Fix unconfigured protocols of UserWallet not found

* Fix Blink API key in wallet seed

* Test send payment on save (except LNC)

This does not include LNC because LNC cannot be saved yet

* Check if window.webln is defined on save

* Create new wallets from templates

* Separate protocols in wallets/lib into individual files

* Use justify-content-start for protocol tabs

and larger margin at the top

* Add LNC to client protocols

* Only return wallets from useWallets

* Query decrypted wallets

* Payments with new wallets

* More wallet logos

* Fix TypeError in useIndexedDB

* Add protocol attach docs

* Fix undefined useWalletRecvPrompt import

* Remove outdated TODOs

* First successful zap to new wallets

* Fix walletLogger imports

* Fix sequences

* the sequences for InvoiceForward and DirectPayment were still starting at 1
* when using setval() with two arguments, nextval() will return the second argument+1 (see https://www.postgresql.org/docs/current/functions-sequence.html)

* Rename ProtocolWallet columns

* Remove more outdated TODOs

* Update wallet indicator

* Fix page reset on route change

* Refactor __typename checks into functions

* Refactor protocol selection into own hook

* Add button to detach protocol

* Refetch wallet on save and detach

* Refetch wallets on change

* Always show all templates

* Refactor WalletLink component

* Also put wallet into forms context

* Remove outdated TODOs

* Use useMemo in wallets hooks

* Passphrase modals

* prompt for password if decryption failed
* add button to reveal passphrase on wallet page

TODO:
* remove button if passphrase was revealed or imported
* encrypt wallets with new key on passphrase reveal

* Fix protocol missing as callback dependency

* Encrypt wallets with new key on passphrase export

* Update 'unlock wallets' text

* Rename wallet mutation hooks

* Remove 'removeWallet' mutation

Wallets are automatically deleted when all protocols are deleted

* Passphrase reset

* Use 110px as minimum width for bip39 words

longest bip39 words are 8 characters and they fit into 103px so I rounded up to 110px.

* Also disable passphrase export on save

* Wallet settings

* Fix wallet receive prompt

* Remove unused parameters from postgres function

* Rename UserWallet to Wallet, ProtocolWallet to WalletProtocol

* Use danger variant for button to show passphrase

* Fix inconsistent imports and exports

* Remove outdated TODOs

* wallet logs

* Remove outdated comment

* Make sure wallets are used in priority order

* Separate wallets from templates in reducer

* Fix missing useCallback dependencies

* Refactor with useWalletLogger hook

* Move enabled to WalletProtocol

* Add checkbox to enable/disable protocol

* Fix migration with prod db dump

* Parse Coinos relay URLs

* Skip network tests if only enabled changed

* Allow IndexedDB calls without session

* Add code to migrate old CryptoKey

* first try to use existing CryptoKey before generating a new one
* bump IDB version to delete old object stores and create new ones
* return IDB callbacks with useMemo
* don't delete old IDB right away, wait until next release

* Fix ghost import error

*Sometimes*, I get import errors because it tries to resolve @/wallets/server to wallets/server.js instead of wallets/server/index.js.

For the files in wallets/server, it kind of makes sense because it's a circular import.

But I don't know why the files in worker/ have this problem.

Interestingly, it only seems to happen with walletLogger imports, so I guess its related to its import chain.

Anyway, this commit should make sure this never happens again ...

* Skip wallets queries if not logged in

* Split CUSTOM wallet into NWC and LN_ADDR

* Migrate local wallets

* Link to /wallets/:id/receive if send not supported

* Hide separator if there are no configured wallets

* Save LNC

* Add one-liner to attach LNC

* Update wallet priorities via DnD

* Wallet logs are part of protocol resolvers

* Fix logging to deleted protocol

* Fix trying to fetch logs for template

* also change type to Int so GraphQL layer can catch trying to fetch string IDs as is the case for templates

* Fix embedded flag for wallets logs not set

* Remove TODO

* Decrease max-height for embedded wallet logs on big screens

* Fix missing refetch on wallet priority update

* Set priorities of all wallets in one tx

* Fix nested state update

* Add DragIcon

* DnD mobile support and refactor

* Add CancelButton to wallet settings

* Remount form if path changes

This fixes the following warning in the console:

"""
Warning: A component is changing an uncontrolled input to be controlled. This is likely caused by the value changing from undefined to a defined value, which should not happen. Decide between using a controlled or uncontrolled input element for the lifetime of the component.
"""

* Support string and object for wallet.image JSON

* Append domain to lightning address inputs

* Remove outdated TODOs

* Add template IDs to wallet JSON

* Fix missing callback dependency

* Implement lightning address save in receive prompt

* Update TODOs

* Fix missing check for enabled

* Pay QR codes with WebLN as anon

* Add logo for NWC

* Fix trying to save logs for template

* Add template logs

* Fix inconsistent margin

* Always throw on missing key

* Remove misleading comment

Wallets are returned even if decryption fails so we can show the unlock page if a wallet is stored as encrypted in the context.

Maybe I should rethink this.

* Check for existing wallets on local wallet migration

* Fix local wallet migration causing duplicates

* Fix protocol reattached on detach due to migration

* Fix form not centered

* Fix ZEUS lightning address domain

* Add placeholder, help, hints etc. to wallet form inputs

* Fix wallet badges not updated

* Remove unused declared variables

* Rename to ATTACH_PAGE

* Fix 500 error if no amount was given to LNURLp endpoint

* Tag log messages with wallet name

* Only skip network tests if we're disabling the wallet

* Rename var to networkTests

* Continue to store key hash in IndexedDB

* Rethink wallet state management

If decryption failed, the function to decrypt the wallets didn't throw but simply returned wallets that were still encrypted.

This was bad because it meant we could not rely on the wallets in the state being decrypted, even though this was the original idea behind the query hooks: hide the details of encryption and decryption inside them.

Because of this, we had to check if the wallets were still encrypted before we ran the wallet migration since we want to check if a protocol already exists.

This commit fixes this by making encryption and decryption always throw (and catching the errors), as well as returning a ready state from hooks. A hook might not be ready because it still needs to load something (in the case of the crypto hooks, it's loading the key from IndexedDB). Callers check that ready state before they call the function returned by the hook.

So now, the wallet migration hook can itself simply check if the hook to encrypt wallets is ready and if the wallets are no longer loading to let callers know if it itself is ready.

Since we also relied on wallets stored as encrypted in the context to show the unlock page, this was also changed by comparing the local and remote key hash.

* Add empty line

* Save new key hash during wallet reset

* Only receive protocol upserts require networkTests param

* Compare key hashes on server on each save

* Delete old code

* Fix card shows attach instead of configure

* Fix empty wallets created during migration

The old schema can contain '' instead of NULL in the columns of wallets for receiving.

* Update reset passphrase text

* Wrap passphrase reset in try/catch

* Fix migrate called multiple times

* Update key hash on migration if not set

* Fetch local wallets in migrate

* Fix missing await on setKey

* Let first device set key hash

* Fix indicator not shown if wallets locked

* Check if IndexedDB is available

* Fix inconsistent WebLN error message

* Disable WebLN if not available

* Remove outdated TODO

* Cursor-based pagination for wallet logs

* Fix log message x-overflow

* Add context to wallet logs

* Wrap errors are warnings in logs

* Rename wallet v2 migrations

* Update wallet status during logging

* Fix wallet logs loading state

The loading state would go from false -> true -> false because it's false when the lazy query wasn't called yet.

* Add wallet search

* Add Alby Go wallet

* Revert "Add Alby Go wallet"

This reverts commit 926c706.

* Fix wallet logs sent by client don't update protocol status

* Fix mutation name

* put drag icon on opposite corner

* Add wallets/README.md

* Fix inconsistent case in wallets/README.md

* Fix autoprefixer warning about mixed support

This warning was in the app logs:

app     | Warning
app     |
app     | (31:3) autoprefixer: end value has mixed support, consider using flex-end instead
app     |
app     | Import trace for requested module:
app     | ./styles/wallet.module.css
app     | ./wallets/client/hooks/prompt.js
app     | ./wallets/client/hooks/index.js
app     | ./wallets/client/context/hooks.js
app     | ./wallets/client/context/provider.js
app     | ./wallets/client/context/index.js

* fix effect of wallet indicators on logo

* Fix deleting wallet template logs

* Use name as primary key of WalletTemplate

* Fix wallet_clear_vault trigger not mentioned in README

* Fix wallet receive prompt

Also remove no longer needed templateId from wallets.json and helper functions

* Use findUnique since name is now primary key

* Merge Alby wallets into one

* Remove unused name parameter from WalletsForm component

* Fix number check to decide if wallet or template

* Update wallet encryption on click, not as effect

* add cashu.me and lightning address logos

* add images

* Use recommended typeof to check if IDB available

* Also check if IDB available on delete

* Use constraint triggers

* Add indices on columns used for joins

* Fix inconsistent CLEAR OR REPLACE TRIGGER

* Attach wallet_check_support trigger to WalletProtocol table

* Update wallets/README.md

* Remove debugging code

* Refactor reducer: replace page with status

* Show 'wallets unavailable' if device does not support IndexedDB

* Remove duplicate ELSIF condition

* Fix hasSendWallet

The useSendWallets hook was not checking if the returned send wallets are enabled.

Since the components that used that hook only need to know if there is a send wallet, I replaced the useSendWallets hook with a useHasSendWallet hook.

* Add Cash App wallet

* fix changes loglevel enum

* Fix key init race condition in strict mode if no key exists yet

* Formatting

* Fix key init race condition via transactions in readwrite mode

* Replace Promise.withResolvers with regular promises

* replace generic spinner with our usual

---------

Co-authored-by: k00b <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant